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ABSTRACT 



Little research has been done into how the results of the Programme for International Student 
Assessment (PISA) affect national educational reform and policy-making. This paper examines the 
normative impact of PISA by investigating how, and the extent to which , national policy actors use PISA 
in policies and practices, to evaluate and improve school-system performance. Drawing on the results of a 
survey of country practices, the study shows that PISA has become accepted as a reliable instrument for 
benchmarking student performance worldwide, and that PISA results have had an influence on policy 
reform in the majority of participating countries/economies. 



RESUME 



A ce jour, rares sont les recherches ciblant I’impact des resultats de I’enquete PISA (Programme 
international pour le suivi des acquis des eleves) sur les reformes et Paction publique dans les pays. Le 
present document examine Pimpact normatif du PISA en cherchant a determiner comment et dans quelle 
mesure les acteurs politiques des pays utilisent le PISA dans Petablissement de politiques et pratiques, afin 
d’evaluer et d’ameliorer les performances de leur systeme d’ education. A partir des resultats d’une enquete 
sur les pratiques des pays, cet etude montre que le PISA est aujourd’hui communement accepte en tant 
qu’ instrument fiable pour evaluer la performance des eleves dans le monde entier et que les resultats au 
PISA exercent une influence reelle sur les reformes politiques dans la majorite des pays et economies 
participants. 
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THE POLICY IMPACT OF PISA: AN EXPLORATION OF THE NORMATIVE EFFECTS OF 
INTERNATIONAL BENCHMARKING IN SCHOOL SYSTEM PERFORMANCE 



Introduction 

This paper seeks to explore the normative impact of the Organisation for Economic Co-operation and 
Development’s (OECD) Programme for International Student Achievement (PISA) on national/federal 
education policy and policy-making processes. PISA is a triennial international comparative study of 
student learning outcomes in reading, mathematics and science - the skills judged to be important for adult 
life in a globalised and knowledge-based society (OECD, 2000; Schleicher, 2007). Whilst originally 
developed as an instrument for OECD countries, PISA has been adopted as an almost global standard, and 
is now used in over 65 countries and economies. As an international benchmarking exercise, PISA is 
designed to monitor outcomes over time and provide insights into the factors that may account for 
differences in performance within and between countries/economies. Pour surveys have been implemented 
so far: 2000, 2003, 2006 and 2009. 

Every three years, the release of PISA results stimulates a global discussion about school reform in 
both international media {e.g. The Economist, The New York Times) and at the national level across many 
OECD and partner countries/economies. ^ Yet, beyond acknowledging that PISA is an interesting news 
story, research on its effects and use within national education reform and policy-making is scarce. This 
paper specifically examines the normative impact of PISA by investigating how, and the extent to which, 
national policy actors use PISA in policies and practices to evaluate and improve the performance of 
school systems. 

The paper is composed of four distinct sections. The first examines the limited extant literature that 
has explored the policy impact of PISA. The second section outlines and justifies the methodology of the 
study. The third section provides a description and analysis of the results. Section four discusses the 
emergent themes and issues. 

Section 1 - PISA and national education policy-making 
^PISA^ shock and policy windows 

Pew countries/economies enacted large-scale policy changes in response to the release of the first two 
PISA survey results in 2001 and 2004. In Germany, the education policy debate and changes in light of 
PISA 2000 were intense {e.g. Ertl, 2006). Confronted with lower-than-expected results in student 
performance, PISA triggered a sustained public debate about education policy and reform that came to be 
known as ‘PISA shock’. The PISA-inspired debate over public education has resulted in a range of 
significant reform measures, including generating national standards and establishing further support for 
disadvantaged students, especially those from immigrant backgrounds (Ertl, 2006). 

A similar reaction to PISA 2000 occurred in Denmark. Egelund (2008), for example, notes that the 
results of PISA 2000 raised serious questions about how the well-funded Danish education system yielded 
only middle-range outcomes, and about why social equity continued to be a problem despite significant 
investment in social welfare programmes. Egelund (2008) notes that whilst the PISA 2000 results had a 
significant impact on the education and political debate, there were no policy changes until after an 
international review of the results and the system was conducted (OECD, 2004). Denmark subsequently 



1 . e.g. Stake (2007) provides an analysis of the media response to PISA in the USA. 
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implemented a range of reform policies, including increased national assessment and evaluation, and 
strategies to target socio-economically disadvantaged and immigrant students (Egelund, 2008). 

Japan has also been identified as a country that experienced a ‘PISA shock’ in the national politics of 
education reform. Japan was a top performer in PISA 2000, yet in PISA 2003 there was a decline in its 
performance resulting in a perceived ‘crisis’ that prompted significant public and political debate on 
education reform. In response to the decline in PISA performance, the Ministry for Education reversed a 
contentious yutori (low pressure) curriculum policy and have enacted changes to national assessment 
practices (Takayama, 2008). 

Kingdon’s (1995) theory of policy agendas may help explain the role of PISA in these large-scale 
reform agendas. Kingdon proposed that an ‘external shock’ could generate a ‘policy window’ for reform. 
This so-called window only opens up the potential for a reform agenda to be established. Yet, this policy 
reform would only occur if and as national actors use the opportunity afforded by lower-than-expected 
PISA results to legitimate the need for large-scale reform. Yet, even in the cases of Germany, Denmark 
and Japan, it is difficult to decipher the direct impact of PISA on the speed or breadth of reforms enacted. 

Comparative national case study investigations have shed further light into the differing national 
responses towards the first two rounds of PISA. Grek (2009) conducted a brief comparative assessment of 
the way PISA 2000 entered national policy-making in Einland, Germany and the United Kingdom, through 
both media attention and subsequent education reforms. Grek (2009) highlights three different responses to 
PISA. Eirst, the ‘PISA surprise’ of Einland: its high performance and international attention surprised the 
nation. Second, the ‘PISA shock’ in Germany, where PISA sparked significant public debate and 
subsequent policy change. Third, the ‘PISA promotion’ of the United Kingdom, where little media 
attention and no reform initiatives followed PISA 2000, yet the results were promoted by the government 
of the day as a sign of strong outcomes. Grek concludes by arguing that PISA can be used nationally as a 
governing resource, as it enables policy actors to justify and legitimise proposed reforms based on 
evidence from internationally comparable data. 

Another recent comparative study^ sought to investigate the capacity for PISA to prompt broad 
education policy convergence across Switzerland, Germany, New Zealand, the United Kingdom and the 
USA (Martens, Nagel, Windzio, and Weymann, 2010). Drawing on expert interviews and policy document 
analysis, the research highlighted the varied national responses to the release of the first round of PISA 
results. Both Switzerland and Germany, for example, initiated significant education policy reforms in 
response to lower-than-expected results (Bieber, 2010; Niemann, 2010). In contrast, the United Kingdom, 
New Zealand and the USA did not show a shift in national education agendas in response to PISA. Knodel 
and Walkenhorst (2010) argue that the United Kingdom’s lack of policy response to its moderate 
performance was due to the significant education reforms enacted in England in the years prior to the 
release of the PISA results. Eor New Zealand, Dobbins (2010) concluded that high performance on PISA 
seemed to reinforce existing policies and thus there was no impetus for substantial change. Einally, 
Dobbins and Martens (2010) propose that, in the case of the USA, substantial domestic evaluation 
programmes may have ensured that there was prior public awareness of its poor educational outcomes and 
thus, unlike Germany, there was no external shock. Martens and Niemann (2010) have further developed 
the notion that a gap between national self-perception of expected performance, and the empirical results, 
may explain the difference between the USA and German PISA responses. 



2 . This research was conducted as part of the Transformations of the State’ project, which sought to 

investigate how different nation-states respond to EU governance through the Bologna Process and OECD 
governance through PISA. The findings of this international project were captured in a recent book 
(Martens et al., 2010). 
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An external evaluation of the policy impact of PISA was conducted in 2008 for the OECD PISA 
Governing Board (Hopkins, Pennock, Ritzen, Ahtaridou and Zimmer, 2008). The evaluation sought to 
explore PISA’s relevance, effectiveness and sustainability, and the unexpected impacts on participant 
countries and economies. The research consisted of a qualitative strand of 548 questionnaire responses 
drawn from a range of education stakeholders from across participating countries/economies. Eive 
qualitative case studies were also conduced in Hong Kong-China, Spain, Canada, Norway and Poland. The 
study covered PISA 2000, 2003 and the initial response to PISA 2006 only. The results revealed that the 
impact of PISA was greater at the national level than at the local government or school level, with policy- 
makers identified as the most significant stakeholder group. The report also highlighted that countries 
increasingly valued the skills assessed in PISA, that the PISA instrument was commonly used for 
monitoring a country’s performance and equity, and that the influence of PISA on policy seemed to be 
increasing over time. Whilst the study did note that a range of policies and initiatives had been undertaken 
in light of PISA, the authors also emphasised that such policy initiatives often displayed a low level of 
policy coherence overall. This paper seeks to build and extend on this broad international analysis of the 
policy impact of PISA over successive survey rounds. 

Many of the aforementioned studies (excluding Hopkins, et al., 2008) have focused on analysing 
whether or not the first round of PISA sparked significant large-scale national reform. This paper goes 
beyond this initial question to explore the potential normative policy impact of PISA resulting from the 
repeated use of PISA nationally and its adoption globally as an international standard. Analysis focuses on 
two areas of inquiry as outlined in figure 1.1: PISA in national policy-making processes; and the use and 
integration of PISA within normative national/federal policy instruments. 

Figure 1.1. Two areas of inquiry 









1. PISA in national/ 
federal policy-making 
processes 



r 






2. The use and integration 
of PISA within normative 
national/federal policy 
instruments 
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•2b. Setting PlSA-referenced or PISA-like cumculum 
standards 

•2c. Setting and monitoring PlSA-based performance 
targets and indicators 
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The study investigates the extent to which PISA, as an international standard of school system 
performance, has been integrated and embedded into national policy and practice. As illustrated in 1.2, 
particular focus is placed on PISA’s role within normative national/federal policies: assessment and 
evaluation, curriculum standards and performance targets. These normative policy instruments function to 
set, measure and reinforce what the education system aims to achieve and its desired improvement 
trajectory. 
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Figure 1.2. Three types of normative nationai/federai poiicy instruments 




Section 2 - Methodology 

2.1. Aims and overall approach 

2.1.1, Research agenda and questions 

This paper explores the normative impact of PISA on national education policies and practices, and 
on policy-making processes. Two areas of inquiry were established to guide the empirical investigation, 
with a combined total of five sub-research areas (la, lb, 2a, 2b and 2c). Specific research questions were 
generated for each area as outlined below. 

Focus area 1: PISA in national policy-making processes 

la. Changes in education policies and practices in light of the PISA evaluation of system performance 

• How influential is the PISA evaluation of national system performance in policy-making 
processes? 

• What changes in educational policies and practices have been made in light of PISA analyses? 

lb. Use of PISA policy implications and reference to high-performing systems in national education 
debates and policy-making 

• To what extent have PISA policy findings been influential in national education debates and 
policy-making? 

• To what extent have high-performing and successfully reforming PISA education systems been 
cited in national policy debates and bi-lateral policy learning? 

Focus area 2: The use and integration of PISA within normative national policy instruments 
2a. The role of PISA in national/federal assessment and evaluation systems 
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• To what extent do national policy-makers see PISA performance as an important indicator of the 
effectiveness of the school system? 

• What role does PISA, among other national and sub-national forms of assessment, play in 
evaluating education-system performance? 

• To what extent is PISA used to monitor the performance of sub-national regions (e.g. states, 
provinces) or specific student groups? 

• How have national learning assessments been linked to PISA or modelled on PISA? 

2b. Setting PISA-referenced or PISA-like curriculum standards 

• To what extent are the competencies outlined in the PISA frameworks and instrument promoted 
in the education system? 

• Have PISA-referenced or PISA-like standards for learning been established? 

2c. Setting and monitoring PISA-based policy performance targets and indictors 

• To what extent do senior policy-makers see improving performance on PISA as a path to 
modernising and improving the national educational system? 

• Have PISA-based national performance targets or indicators been established? What is the focus 
and nature of these targets? 

2.1.2. Data collection 

An expert questionnaire, completed by members of the PISA Governing Board (PGB), was the central 
method of data collection for this study. An online questionnaire and overview of the study’s interests were 
sent via e-mail, in July 2011, to the PGB representatives from the 65 countries/economies that participated 
in PISA 2009. An identical Word document e-mail version of the survey was also available and requested 
by six representatives. These national representatives are formally appointed by their governments. Their 
official role is to oversee the governance of the PISA project and link the international work of PISA back 
into national ministries of education. They are invaluable informants as they are most likely to understand 
the extent to which and manner in which PISA is used within their home countries/economies. The purpose 
of the questionnaire was to gain expert insights on how PISA has been used within national policies and 
practices across a broad array of contexts at the national/federal and senior policy-maker levels. 

The construction of the questionnaire was modelled on the national/federal policy areas of particular 
interest to the research questions. The survey was written in English, as this is the language in which PGB 
meetings are most commonly conducted. 

2.1.3. Sample 

A combined total of 37 representatives from OECD countries/economies and partner 
countries/economies responded to the questionnaire. OECD countries that participated in the data 
collection included: Australia, Austria, Belgium (Erench and Elemish Communities), Canada, Chile, 
Denmark, Estonia, Einland, Erance, Germany, Greece, Hungary, Ireland, Israel, Italy, Japan, Korea, 
Luxembourg, Mexico, the Netherlands, Norway, Poland, Portugal, the Slovak Republic, Slovenia, Spain, 
Sweden, Turkey, the United Kingdom (England, Scotland and Wales), and the United States of America 
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(USA). The partner countries/economies Indonesia, Latvia, Singapore and Hong Kong-China also 
participated in the data collection. 

Separate responses were obtained from the French and Flemish Communities of Belgium, and also for 
the English, Scottish and Welsh education systems of the United Kingdom. Consequently, these responses 
were analysed and reported separately. A total of 37 different education systems provided separate 
responses to the questionnaire.^ For ease of expression, the term countries/economies is used to refer to all 
distinct education systems that provided a response to the questionnaire. 

2.1.4. Limitations of the research 

This research methodology has a number of limitations. First, the results draw on only one respondent 
per country/economy."^ It is possible that a broader range of respondents from each country would have 
developed a more comprehensive analysis of the use of PISA. Yet, this brief exploratory study aimed to 
identify, in broad terms, the uses of PISA across multiple countries/economies, without a deep analysis of 
the way PISA plays into complex policy debates and decisions. The PGB representatives are best placed to 
provide a snapshot of the connection between PISA and national policies and practices. Qualitative case 
studies of specific countries/economies could be used to further explore the themes raised in this study, and 
would benefit from seeking expert opinions from policy-makers at different levels of government as well 
as from academic institutions. 

Second, only a small number of partner countries/economies took part in the questionnaire. It is 
possible that policy-makers in some of these systems may use PISA differently than how those in OECD 
countries do, due to limited national capacities in assessment and evaluation. 

Third, some of the questionnaire items required respondents to make subjective judgments regarding 
the extent of influence or role of PISA in policy-making. Comparisons between country/economy 
responses for these items should be drawn with care, as respondents may have had different interpretations 
of these terms and ratings. 

Fourth, the questionnaire consisted of both multiple-choice and extended-response questions. Whilst 
all 37 respondents completed the multiple-choice questions, there was substantial variation in the 
additional information provided by respondents in the extended-response questions. This limits the number 
of countries/economies for which specific examples of policies and practices can be provided in the results 
section below. 

Section 3 - Results 

This section examines the results of the PGB questionnaire as they relate to this study’s five specific 
areas of research: 

3.1. Changes in educational policies and practices in light of PISA analysis of system performance; 

3.2. Use of PISA policy implications and reference to ‘high-performing’ systems in national 
education debate and policy-making; 



3 . Questionnaires were also received from Uruguay and Brazil. Yet, due to the incomplete nature of these 

responses, these were not included in the final analysis. 

4 . Responses from Sweden and Korea made clear that multiple individuals had taken part in contributing to 

the final responses that were provided. 
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3.3. The role of PISA in national/federal assessment and evaluation policy and practices; 

3.4. Setting and revising curriculum standards; and 

3.5. Setting and monitoring performance targets and indicators. 

3.1. Changes in educational policies and practices in light of PISA results 

3.1.1, Broad analysis of PISA policy impact 

Respondents were asked how influential specific PISA results and analyses have been in informing 
the policy-making process at the national/federal level in the country they represent. Respondents from 17 
countries/economies judged that PISA had been ‘very’ influential, and respondents from a further 11 
countries/economies judged PISA to be ‘moderately’ influential. The respondents from England-UK, 
Denmark, and Japan rated PISA as ‘extremely’ influential, whilst respondents from Einland, Prance, 
Indonesia, Luxembourg and Turkey rated PISA as ‘not very’ influential. The respondent from Belgium 
(Plemish Community) selected ‘don’t know.’ 

Beyond this overall judgment of PISA’s influence in national policy-making, respondents were asked 
to indicate which specific aspects of PISA results, across any survey round, have led to/inspired changes in 
policy and practice in the country/economy they represent. The PISA result categories included: a) overall 
international rank; b) reading performance; c) mathematics performance; d) science performance; e) trend 
performance; f) equity; g) student interest, engagement, motivation and attitudes; and h) other issues. The 
three response options were ‘Yes’, ‘Partly’ and ‘No’. These results are shown in Table 3.1.1. 

Respondents from 26 different countries/economies indicated ‘yes’ to at least one category of PISA 
results. Australia was the only country that did not select either a ‘yes’ or ‘partly’ for at least one of the 
PISA result categories. Respondents indicated that the overall international rank had led to/inspired 
changes in 19 countries/economies and partially led to change in a further 11 countries/economies. 
Respondents from 20 countries/economies indicated that results of the PISA reading assessment had 
prompted policy changes; 16 countries/economies cited performance on the PISA mathematics assessment 
as leading to policy changes; and 12 countries/economies cited results of the PISA science assessment as 
leading to policy changes. Only the respondents from Australia and Prance answered that ‘no’ change had 
occurred in response to PISA reading performance results. 
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Table 3.1.1 

Question 18. Which aspects of your country’s PISA results (across any survey roun), have led to/inspired changed in 

policy or practice in your country? 





Yes 


Partly 


No 


A. Overall internaiional rank 


51.4% (19) 


29.7% (11) 


18.9% (7) 


B. Reading performance 


54.1% (20} 


40.5% (15) 


5.4% (2) 


C. Mathematics performance 


43.2% (16) 


43.2% (16) 


13.5% (5) 


D. Science Performance 


32.4% (12) 


54.1% (20) 


13.5% (5) 


E. Trend performance 


40.5% (15) 


29.7% (11) 


29.7% (11) 


F. Equity 


29.7% {11} 


43.2% (16) 


27.0% (10) 


G. Student interest, engagement 
motivation and attitudes 


8.1% (3) 


56.8% (21) 


35.1% (13) 


H. Other issues (please state 
below) 


8.1% (3) 


8.1% (3) 


83.8% (31) 



Respondents from 15 countries/economies indicated that trend performance had led to policy changes, 
and respondents from 11 countries/economies noted that equity issues had led to changes. Only 
respondents representing Austria, Korea and Japan indicated that ‘student, interest, engagement, 
motivation and attitudes’, as revealed in PISA results, had led to changes in policy and practice. 

An index was formed to summarise the breadth of policy impact across PISA results for each 
country/economy. The breadth of policy impact index was constructed by allocating each country 2 points 
for each ‘Yes’, 1 point for each ‘Partly’ and 0 points for each ‘No’ across the seven PISA result categories 
(a-g in Table 3.1.1). A higher score on this index represents policy responses to a broader range of PISA 
result categories in that country/economy. The combined total for each country/economy is shown in Table 
3.1.2. To aid in the interpretation of the results, the table includes the respondents’ judgment of how 
influential PISA had been in national policy-making processes, and also shows the PISA 2009 reading, 
mathematic and science scores, indicated as above OECD average, OECD average or below OECD 
average (OECD, 2010a). Table 3.1.2 suggests that it would be wrong to assume that EISA’s impact on 
policy-making is necessarily greater in countries/economies that perform below the OECD average. 

High-performing countries such as Korea and Japan have enacted reforms in response to a large range 
of PISA results. In contrast, a large group of countries that perform below or at the OECD average, 
including Italy, Indonesia and Erance, indicated very little policy change in response to PISA. 
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3.1.2. The role of PISA in policy-making processes and policy responses to PISA. 

Respondents were asked to provide further details on the role PISA results play in policy-making 
processes. Analysis of the results shows several key areas of influence: 

• PISA results set the agenda for policy dialogue among policy-makers and experts (e.g. Austria, 
Korea). For example: 

Korea - “Based on the results of PISA, Korean policy makers have seriously considered the facts 
that the gender difference has been increased and the student ration of high performances has 
been lower than the other high-performing countries'' 

• National policy actors use PISA results as evidence to argue that national performance needs 
improvement overall, or in a specific area. For example: 

England-UK - “Overall ranking and trend performance in PISA demonstrates that other 
countries are improving at a faster rate and overtaking us, therefore demonstrating the necessity 
for the reforms set out in last year's white paper." 

• PISA indicators define performance and equity issues {e.g. variance between schools). 

• PISA results are used as one of multiple indicators to inform policy decisions and development 
{e.g. Australia, Scotland-UK, Hong Kong-China). The respondent from Scotland-UK noted: 

“PISA is a key measure, in particular our decline in the international rankings and the strong 
socio-economic background effect, but policy decisions are taken on the basis of a balance of 
international and national evidence." 
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Table 3.1.2. 



Country/Economy 


Breadth of 
policy impact 
index 


Informing 

poiicy-making 

process 


PiSA 2009 
reading 


PiSA 2009 
mathematics 


PiSA 2009 
science 


Republic of Korea 


14 


Very 


Above 


Above 


Above 


Austria 


13 


Very 


Below 


Average 


Below 


Hungary 


13 


Very 


Average 


Average 


Average 


Japan 


13 


Extremely 


Above 


Above 


Above 


Israel 


13 


Very 


Below 


Below 


Below 


Germany 


12 


Very 


Average 


Above 


Above 


Sweden 


12 


Very 


Average 


Average 


Below 


Poland 


12 


Very 


Above 


Average 


Above 


Ireland 


11 


Very 


Average 


Below 


Above 


Latvia 


11 


Very 


Below 


Below 


Below 


England-UK 


11 


Extremely 


Average 


Average 


Above 


Denmark 


10 


Extremely 


Average 


Above 


Average 


Greece 


10 


Very 


Below 


Below 


Below 


Norway 


10 


Very 


Above 


Average 


Average 


Slovak Republic 


10 


Very 


Below 


Average 


Below 


Estonia 


9 


Very 


Above 


Above 


Above 


Wales-UK 


9 


Very 


Below 


Below 


Average 


Chile 


8 


Moderately 


Below 


Below 


Below 


Luxembourg 


8 


Not very 


Below 


Below 


Below 


Mexico 


8 


Very 


Below 


Below 


Below 


Netherlands 


8 


Moderately 


Above 


Above 


Above 


Turkey 


8 


Not Very 


Below 


Below 


Below 


Finland 


7 


Not Very 


Above 


Above 


Above 


Hong Kong-China 


7 


Moderately 


Above 


Above 


Above 


Scotland-UK 


7 


Moderately 


Above 


Average 


Above 


Slovenia 


6 


Very 


Below 


Above 


Above 


Spain 


6 


Moderately 


Below 


Below 


Below 


USA 


6 


Moderately 


Average 


Below 


Average 


Belgium (Flemish 
Community) 


5 


Don’t Know 


Above^ 


Above^ 


Above^ 


Singapore 


5 


Moderately 


Above 


Above 


Above 


Belgium (French 
Community) 


4 


Very 


Above^ 


Above^ 


Above^ 


Indonesia 


4 


Not very 


Below 


Below 


Below 


Italy 


4 


Moderately 


Below 


Below 


Below 


Portugal 


4 


Moderately 


Average 


Below 


Below 


Canada 


3 


Moderately 


Above 


Above 


Above 


France 


1 


Not very 


Average 


Average 


Average 


Australia 


0 


Moderately 


Above 


Above 


Above 



^ Is the overall Belgium national performance 
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• PISA results are used as an external trigger for large-scale public debate on education and 
relatively rapid policy dialogue and policy change (e.g. Germany and Denmark). For example, 
the respondent from Germany stated: 

''The PISA-2000 results were rather disappointing and had a high impact on educational policy. 
People often refer to this as some sort of beneficial PISA shock . . . The German States started 
many programmes and projects at the regional/local level as well as at the federal level to 
approach the problems that PISA had revealed'' 

Respondents were also asked to provide details of the nature of the changes to policy or practice and 
explain how PISA results were connected to these changes. An analysis of the open responses revealed a 
broad range of policies and initiatives, including: 

• Formation, expansion and improvements to national/federal assessment and evaluation systems 
(further examined in Section 3.3); 

• Revision of curriculum standards, often to include and emphasise PISA-like competencies {e.g. 
Ireland, Germany, Greece and Norway) (further examined in Section 3.4); 

• Setting and monitoring performance targets and indicators (further examined in Section 3.5); 

• Strategies aimed at specifically improving reading/literacy performance {e.g. Japan and Ireland); 

• Promoting equity through school financing {e.g. Austria, Belgium [Flemish] and Hungary); 

• Postponing student tracking decisions {e.g. Belgium [French Community]); 

• Strategies to monitor and improve the proportion of poor or top PISA performers {e.g. Korea and 
European Union); 

• Strategies and discussions on how to improve student engagement and attitudes {e.g. Austria, 
Japan and Korea); 

• Promoting competency-based teaching and learning {e.g. Hungary); and 

• Additional resources for infrastructure development for natural sciences {e.g. Fatvia). 

3.2. Use of PISA policy findings in national/federal education debate and policy-making 

PISA produces policy findings through both a quantitative analysis of the policy factors associated 
with high performance and equity across systems,^ and through further case-study analysis of the reforms 
and policies implemented in high-performing or improving PISA countries.^ 



5 . Reported in both the PISA Initial Report for each survey round and through additional thematic reports. 

6 . For example, the recent publication. Strong Performers and Successful Reformers: Lessons from 

PISA for the United States (2010b). 
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3.2.1. Influence of PISA policy findings 

PISA aims to draw policy lessons and implications for the factors that are associated with high 
performance and equity. Respondents were asked to rate whether or not six areas of PISA policy analysis^ 
have been influential in national policy-making processes. Response options were ‘Yes’ and ‘No’. As 
shown in Table 3.2, ‘Assessment and accountability’ policy findings have had the broadest influence, with 
respondents from 29 countries/economies judging that this finding had been influential in national policy- 
making processes. This finding is well supported by the reported changes to national/federal assessment 
policies and practices outlined in Section 3.2. Between 11 and 13 respondents rated that the five other 
major policy areas had been influential in their country/economy. Some respondents emphasised that PISA 
policy findings had confirmed the countries ’/economies’ previously determined policy direction rather than 
contributing to new policy decisions {e.g. Sweden and Singapore). 

Table 3.2 

Question 20. Which areas of PISA policy analysis have been influential in national policy-making processes? 



Policy areas 


Yes’ 


a. Assessment and accountability 


29 


b. Learning environment 


13 


c. Early childhood education 


13 


d. Resource invested and allocation 


12 


e. Student selection and tracking 


11 


f. Governance [e.g. autonomy, choice, private/public). 


11 



One policy implication of interest is selection and tracking. Whilst these processes are not relevant to 
all education systems, PISA findings have played an influential role in policy discussions on this issue in 
1 1 countries/economies. Of particular interest among these countries is the group of European countries 
that has a history of selection and tracking. These include Austria, Belgium, Denmark, Germany, Hungary, 
Luxembourg, Poland and the Slovak Republic. The respondent from Chile also noted that PISA findings 
related to selection and tracking have been cited during discussions of that country’s General Education 
Law. 

3.2.2. Reference to and learning from high-performing PISA countries/economies 

Respondents from 24 countries/economies judged that the policies of, or references to, high- 
performing or successfully reforming PISA systems have been influential in policy-making processes. 
Some respondents identified a more general influence of PISA high-performing systems on policy: 

Scotland-UK - “7n general the practices of high-performing countries are part of the evidence 
based used in developing policies. No Individual country of country’s policies has been 
specifically influential.'” 

Australia - ‘Australia is strongly interested in PISA-identified high performing and successfully 
reforming systems^ and considers these results when formulating educational policy. However, 



1 . These policy areas were derived from the key sub-sections of the PISA 2009 initial report, Volume IV 

(OECD, 2010c). 
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Australia recognises that it is not always possible or desirable to introduce specific policies from 
other countries, as the country-specific context must be taken into accounts 

The systems of Finland, Korea, Shanghai-China, Singapore, Canada, Australia and New Zealand were 
frequently cited as being influential in policy-making processes. Many respondents noted the influence of 
more than one system. For example, the respondent from Chile wrote: 

''The experiences from several countries have been considered when developing different 
educational policies. For example, Finland is quoted in regard to equity, high performance, 
teacher training and absence of a high stake assessment system. The United Kingdom was 
reviewed for curriculum, institutional organization and, recently, for assessment consequences. 
And some US states .... Massachusetts for assessment and accountability. Other countries 
considered as well are Australia, Korea, New Zealand and Spain.'' 

Finland was the most commonly listed influential country/economy. Respondents from Denmark, 
Norway and Sweden noted a broad interest in Finnish policies as they seek to understand the performance 
differences between their Scandinavian systems. For example: 

Sweden - "In Sweden, much attention has been given to Finland and the Finnish success in PISA, 
given the many common elements of our education systems. Factors of relevance are the teacher 
education, support of individual students, special education, early intervention when students 
need extra resources, among others." 

A diverse group of 1 1 additional countries/economies also emphasised the influence of Finland. This 
group consists of the French and Flemish Communities of Belgium, Chile, Greece, Japan, Fatvia, Poland, 
Singapore, the Slovak Republic, Slovenia, and Spain. Open-responses made clear that a number of 
countries/economies had a particular interest in the Finnish approach to teacher recruitment, training and 
early intervention for student support. For example: 

• Spain - “Finnish ways to select the best candidates to become teachers . . . initial teaching training 
policies are currently used to inner discussions about our own initial teaching training system.” 

• Greece - “Finnish education system has influenced the policy debate about how the quality of the 
Greek education system might further improve. Moreover, it has also informed policy-making 
processes concerning educational reform in Greece. Precisely, Finnish curricula have provided 
useful inputs (especially regarding PISA-like competencies) in the Greek curricula revision 
processes.” 

• Slovak Republic - “Finland is the most inspiring system for our educational policy, mostly in the 
spheres of teacher appraisal, the position of teachers in society, school financing, autonomy of 
schools and freedom of creation of curricula.” 

For some countries/economies, their PISA peers of interest were high-performers that achieved 
particularly well in a specific area. The respondent from the French Community of Belgium cited the 
progress among Polish students. The respondent from Korea noted that: 

"Although Korea has showed strong performance in PISA reading, the reading policy and 
curriculum of the countries/economies which indicate the higher percentage of top-level students, 
such as New Zealand and Australia, are considered to be further investigated" . 

The respondent from Singapore stated that the: 
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''examples of Finland and Shanghai in supporting weak performers or weak schools are instructive 
as we review our own strategies. ” 

3.3. The role of PISA in national/federal assessment and evaluation policies and practices 

Respondents were asked to rate the extent to which policy-makers in their country/economy see PISA 
performance as an important indicator of the effectiveness of the school system. As illustrated in Figure 

3.3, only Israel and England-UK rated PISA as ‘extremely’ important in this regard. Respondents from 24 
countries/economies rated PISA as ‘very’ important and nine respondents rated PISA as ‘moderately’ 
important. Turkey and the partner country Indonesia rated PISA as ‘not very’ important as an indicator of 
the effectiveness of the school system. 



Figure 3.3. 







Response 

Percent 


Response 

Count 


Extremely 


□ 


5.4% 


2 


Very 


1 


1 64.9% 


24 


Moderately 


1 1 


24.3% 


9 


Not very 


□ 


5.4% 


2 


Not at all 




0.0% 


0 


Don't know 




0.0% 


0 



Q7. To what extent is PISA performance seen by policy-makers in your country as an important indicator 
of the effectiveness of the school system? 

An analysis of the responses to open-ended questions on this subject highlights six key ways the PISA 
instrument had been used: 

1. Provided a system evaluation for countries/economies that did not previously carry out 
national/federal assessments; 

2. Led to the formation, or increased the scope, of a national/federal assessment system; 

3. Complemented national data and validated national results against an international benchmark; 

4. Used to conduct within-country/economy monitoring of sub-national regions or student groups; 

5. Used as a best-practice ‘model’ or ‘guide’ for the formation and adaption of national/federal 
assessment policies and practices; and 

6. Used to evaluate the effectiveness of educational reforms. 
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3.3.1. Provided a system evaluation for countries/economies that did not previously carry out 
national/federal assessments 

Large-scale national assessments other than PISA were in operation in all countries/economies except 
for Greece, Latvia and Poland. Lor these countries/economies, PISA provided an important tool for 
evaluating the system. The respondent from Greece explained: 

‘'PISA has provided policy-makers with useful information and tools to improve the quality and 
efficiency of the existing education system in Greece. More specifically, PISA results were used 
as a basis both for detecting systemic weaknesses of our existing education system and informing 
the revision of compulsory education curricula. ” 

3.3.2. Led to the formation, or increased the scope, of a national/federal assessment system 

Respondents from 24 countries/economies indicated that PISA had influenced the development of 
new elements of a national/federal assessment strategy {e.g. the formation of new national assessments or 
adaptation of existing assessments). Respondents were also asked to provide details of these changes and 
judge the extent to which these changes had been a direct consequence of the country’s/economy’s 
engagement with PISA. 

In many countries/economies, including Austria, Germany, Hungary, Ireland, Japan, Luxembourg, 
Norway, Poland and the Slovak Republic, PISA was judged to have played a significant role in 
highlighting the need to initiate further national/federal assessments in order to strengthen the evaluation 
and monitoring of student learning outcomes. Examples of country/economy responses include: 

• Slovak Republic - ‘Under the influence of PISA, we implemented new national measurements of 
reading and mathematics, as a direct consequence of poor results of our country in PISA 2003 
and 2006 cycles. ” 

• Japan - “It had been decided to introduce a national assessment of student performance after the 
release of the PISA 2003 results in 2004, and actually implemented since 2007. PISA-type 
assessment items are being used in the national assessment. 

• Ireland - “PISA has helped to identify the need to better assess the learning outcomes of students 
at second level. However, the assessment instruments, to be implemented nationally in 2012 in a 
systematic manner, are standardised tests and as such are not directly influenced by PISA. 
However, the monitoring and evaluating function of such tests is.” 

In other countries/economies, involvement in PISA was judged to have led to increasing the scope of 
national assessments across subjects or years (e.g. Korea and Sweden). The respondent from Sweden 
noted: 



“There has been an increase in the number of subjects tested within the national assessment 
system. It is likely that the Swedish PISA results in science contributed to the introduction of 
national tests in biology, chemistry and physics for 12- and 15-year-olds as well as the 
development of diagnostic material in science for the earlier years in compulsory school. The 
introduction of national tests in mathematics in grade 3 (9-year-olds) is also partly another result 
of Sweden's PISA engagement. ” 
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3.3.3. Complemented national data and validated national results against an international benchmark 

Respondents from some countries/economies emphasised that PISA results complement national data 
derived from national/federal assessments. For a group of high-performing countries/economies PISA 
analysis is used to compare and validate data from their own national and sub-national assessments, even 
though national data remained more influential in system evaluations. Respondents emphasised that the 
PISA assessment of competencies complemented national assessments, which often focused on testing 
knowledge outlined in national/state curricula. PISA also allowed policy-makers to assess important 
disaggregated results, such as low and high achievers, the impact of socio-economic and immigrant 
background, and attitudes towards learning. Examples of country/economy responses included: 

Canada - ''PISA is used by Canadian provinces as a complementary indicator to monitor overall 
system performance as well as tracking results over time for specific subpopulations. It is also 
used to monitor the change in key variables associated with learning (e.g. reading enjoyment). ” 

Hong Kong-China - "The functional approach of measuring application of knowledge to daily 
life is consistent with the national assessment in Chinese^ English and Mathematics. ” 

Singapore - "PISA data complements national assessment data to inform us about the 
effectiveness of the education system. This includes information about top and low performers; 
and the different types of schools . '' 

Spain - "PISA is currently used as a means of comparing and contrasting our data generated by 
national and regional assessment studies. PISA is an important referent. ” 

Some countries/economies indicated that policy-makers have sought to more effectively 
accommodate and integrate PISA within their overall assessment strategy. Changes cited included taking 
into account the timing of PISA when planning the subjects/domains to be assessed by other national 
programmes {e.g. Belgium [Flemish]), and seeking to minimise the burden of testing on both schools and 
assessment authorities (Finland). The respondent for Canada noted: 

"The national assessment program was redesigned in 2007 to better complement PISA in terms 
of schedule, target population and subject areas covered. ” 

Respondents were asked if existing national or sub-national assessments had been ‘linked’ to PISA 
by, for example, including PISA items in national or sub-national assessments. Respondents in eight 
countries/economies indicated that some form of linking had occurred. Analysis of the open responses 
revealed a variety of linkages: comparing the performance of the same cohort of students on a national 
assessment and PISA without statistical linkage {e.g. Slovenia); inclusion of some PISA items or PISA-like 
items in national assessments {e.g. Austria); and incorporating PISA international benchmarks within their 
current national or sub-national assessments {e.g. Canada). Examples included: 

• Canada - "Validation of provincial performance levels in British Columbia and Ontario.^ 

• Chile - "We administered our previous year national assessment to the PISA ninth grade PISA 
2009 sample. In 2006 we took the grade option and it coincided with our national assessment. 
This was done with the idea of linking national and PISA results, which is part of our future 
plans. ” 



8 . See - Linking provincial student assessments with national and international assessments, 

http://dsp-psd.pwgsc.gc.ca/Collection/Statcan/81-595-MIE/81-595-MIE2003005.pdf 
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• Mexico - ''We linked PISA 2006 results with ENLACE (Ministry of Education national 
assessment) in 2008. ” 

• USA - "Three states have embedded PISA items in their state assessments to help with setting 
proficiency levels in their state assessments . '' 

3.3.4. Used to conduct within-country/ -economy monitoring of sub-national regions or specific student 
groups 

Respondents from 14 countries/economies (Australia, Belgium [Flemish Community], Canada, 
Denmark, Estonia, Finland, Israel, Italy, Mexico, Poland, Slovenia, Spain, Singapore and the United 
Kingdom) indicated that PISA was used by national actors to monitor the performance of sub-national 
regions {e.g. states, provinces) or specific student groups. To do so, countries/economies must often choose 
to test a larger student sample that is stratified by region or particular student groups. 

The analysis revealed that PISA is used as a monitoring policy tool in a group of large federal systems 
where education is mostly controlled at the state/provincial level. These countries include Australia, 
Canada, Germany and Mexico. Australia reports on the PISA performance of six states and two territories 
annually in the Report on Government Services. Since 2003, Mexico has over-sampled in order to obtain a 
representative sample of each of the 32 federal entities (also attempted to gain a better distribution between 
secondary/lower secondary and between urban/ rural populations). Other examples included: 

• Canada - "Since education is the exclusive responsibility of provinces and territories in Canada, 
each province monitors PISA results very closely to validate their own provincial data and other 
national level results. ” 

• Germany - "PISA was used to monitor the performance of the 16 federal Lander in 2000, 2003 
and 2006. Since the initial of a national assessment in 2009, PISA has no longer been used for 
this level of monitoring. ” 

Drawing on both the results of the questionnaire and PISA sampling information in the PISA 2009 
initial report (OECD, 2010), it is clear that student samples are also stratified at the regional level in Spain 
(15 regions, adjudicated), Italy (21 regions, non-adjudicated), and the United Kingdom into Scotland 
(adjudicated), England, Northern Ireland and Wales (non-adjudicated). Yet the extent to which these 
regional results are used for monitoring is beyond the scope of this study. The respondent from Finland 
noted that student results were monitored at the regional level, and in PISA 2009 two regions out of nine in 
Austria participated, but this is not set to continue for PISA 2012. In Denmark, additional instances of 
PISA testing have been used to monitor sub-national groups and students: 

^^PISA is used for extra testing of bilingual students and one Danish municipality has used PISA 
3 times for testing all of the students in the municipality (9th grade). ” 

In Poland, PISA has been used to assess older cohorts (16- and 17 -year-olds). 

Analysis of the open responses found that monitoring of performance by student language group has 
been conducted in Finland (Finnish-speaking and Swedish-speaking) and Israel (Hebrew- and Arabic- 
speaking). Sweden over-sampled students with a foreign background in PISA 2000 and PISA 2003 in 
order to further analyse this sub-group. Yet this was not continued for PISA 2006 and PISA 2009. 
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3.3.5. Used as a best-practice ^modeV or ‘ guide ^ for the formation and adaptation of national/federal 
assessment practices 

Open-ended responses from respondents highlighted that PISA has been used as a best-practice 
‘model’ or ‘guide’ for the design of new national assessments and adaptations to existing assessments. 
Examples included: 

• Hungary - ''The Hungarian Assessment of Basic Competencies is not linked to PISA by anchor 
items, so not exactly the same scale is used, but the framework of the national assessment system 
is based on the model of PISA. ” "PISA has a great impact on the design and the framework of 
the Hungarian Assessment of Basic Competencies. Data analysis and reporting is also similar to 
the methods established by PISA. ” 

• Korea - "Test developers and subject area experts are trying to benchmark and reflect the item 
type, assessment frameworks, as well as the content of PISA assessment to some extent. ” 

• Slovak Republic - "Our national measurements at the level of primary schools (testing of ninth- 
grade students) have been influenced by PISA evaluation framework in following spheres - 
structure of tests, cognitive levels, assessment of questions etc. The reason of this linking is that 
our national measurements should evaluate skills that are essential for everyday life and lifelong 
learning in the similar meaning as PISA has. ” 

• Italy - "The frameworks used as a basis for national assessments are specifically modelled on 
PISA and other international survey frameworks. ” 

Respondents from Erance and Chile emphasised the role of PISA in guiding technical and 
methodological changes to the way national assessment are conducted: 

• Prance - "Since 2003, national assessments on large samples use the methodology of PISA to 
establish competences scales. It's a direct consequence of France's engagement with PISA. ” 

• Chile - improving procedure, manuals, item construction, statistical analysis and keeping 
records. ” 

In some high-performing countries/economies, the national assessment goals and practices were 
judged by respondents to be already consistent with the PISA instrument. Consequently, the model of 
PISA provides a departure point for dialogue, validation and innovation in national assessments. Por 
example, Singapore: 

"Since the late 1990s, Singapore has been making changes to the item types in our national 
exams to test for higher-order thinking. PISA items have provided us with good ideas as we 
consider how to assess the learning outcomes that are valued in the Singapore education 
system.” ..."How PISA implements computer-based assessments informs our own efforts in 
computer-delivered assessments in our national examinations. ” 

3.3.6, Used to evaluate the effectiveness of educational reforms 

Respondents from 10 countries/economies indicated that PISA was used to monitor or evaluate the 
effectiveness of specific or large-scale policy reforms. Examples included: 
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• Hong Kong-China - ‘'English as medium of Instruction Policy, Parental Involvement Policy and 
Educational Decentralisation Policy. ” 

• Poland - “PISA is important in monitoring the massive educational reform which started in 
September 1999 on ISCED 1 and 2 leveV’ and in 2001 for ISCED 3 level. 

• Singapore - “Along with other studies, PISA is used to provide an indication of the effectiveness 
of our initiatives to promote critical and inventive thinking; help under-achievers; and maximise 
the potential of students. ” 

Other respondents indicated that PISA would be used as an indicator of the effectiveness of recent or 
soon-to-be-enacted reforms. For example, for Ireland, there is an expectation that later rounds of PISA will 
“be used to evaluate the effectiveness of the recent national literacy and numeracy strategy and curricular 
reforms.'' Similarly, the respondent for Austria indicated that PISA 2012 would be used to evaluate the 
effectiveness of reforms initiated after 2007, which could not be reasonably be assessed by PISA 2009 due 
to the time lag of reform effects. Other relevant country/economy examples included: 

• England-UK - “PISA 2009 performance will be an informal baseline from which to measure the 
success of reforms currently being put in place. ” 

• Israel - “PISA will serve as one indicator for the effectiveness of a program set to boost 
scholastic achievement in middle schools. ” 

• Mexico - “An improvement in PISA results will be one of the indicators considered for the 
assessment of the Education Sector Programme (PROSEDU) 2007-2012 Objective 1. The results 
of PISA 2009 and an instrument PISA-type were used on the evaluation of ISCED reform. ” 

3.4. Setting and revising curriculum standards 

Respondents were asked to rate the perceived level of alignment between PISA competencies and the 
goals of the education system as set by the government. Respondents from 19 countries/economies judged 
the two to be Aery’ aligned and a further 15 judged them to be ‘moderately’ aligned. The respondent from 
Japan noted the two were ‘extremely’ aligned, whilst the respondents from Greece and Luxembourg 
determined that PISA competencies and the goals of their education systems were ‘not very’ aligned. 

Respondents were asked whether national or sub-national curriculum standards have been created or 
adjusted with reference to PISA or PISA-like competencies. Questionnaire respondents from 16 
countries/economies judged that national or sub-national curriculum standards had been created or adjusted 
(a ‘yes’ response). Respondents were also asked to provide details of the nature of these standards, and 
make a judgment as to the extent to which these changes were a direct consequence of the country’s 
engagement with PISA (Question 4). Analysis of the open-ended responses highlighted the following four 
roles and uses of PISA in setting standards and revising curricula: 

i. Curriculum standards and goals already aligned with PISA competencies, so no pressure for 
change exists (Australia, Canada, Singapore, Sweden and Hong Kong-China). 

a. Policy-makers and experts have assessed PISA frameworks among other international and 
national examples in discussing and revising curricula and standards. 

- Ireland - “National comparisons of curricula with the PISA frameworks and assessment 
have fed into the overall influence of the PISA results." 
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- USA - ‘‘PISA has been assessed, along with other frameworks, in the formation of the 
new Common Core Standards.'' 

- Slovenia -“Indirect influence of PISA assessment and PISA-like competencies in the 
setting and interpretation of national curriculum standards." 

- Hong Kong-China - “PISA framework has been widely discussed by the curriculum 
experts of the government." 

Hi. Revision of curriculum standards, with an explicit attempt to align with PISA frameworks or 
include PISA-like competencies. 

For example, curriculum developers/reviewers have tried to reflect PISA competencies in national 
curricula in Korea (e.g. revision of science curriculum standards) and Mexico (lower secondary level). 
Other examples included: 

- Chile - “PISA 's reading framework had a great impact in curriculum reform and in the 
content and performance standards.""^ 

- Japan - “After the release of the PISA 2003 results in 2004, it was decided to revise the 
national curriculum (courses of study) to incorporate the fostering of PISA-like 
competencies and to improve the student academic performance." 

- Slovak Republic - “Key competencies in reading, mathematics and science characterised 
in detail by PISA were implemented into national curriculum standards in the Slovak 
Republic. These competencies did not have a tradition in our previous educational 
system." 

- Spain - “The LOE (Organic Law of Education, 2/2006, 3rd May) acknowledges basic or 
key competencies as a fourth element of the curriculum ( objectives, content, evaluation 
criteria and competencies). The influence of PISA literacies on this innovation cannot be 
denied." 

- Norway - “In the latest curricular reform in Norway - The Knowledge Promotion, 
LK2006 - basic skills in reading and mathematics are included in all subject curricula." 

Responses from Greece and Luxembourg, the two countries/economies that rated the least alignment 
between PISA competencies and the current goals of the education system, indicated movement towards 
reforms that will include PISA competencies within national curriculum standards. The respondents noted: 

- Greece - “Greek students’ low achievement in the PISA survey have inspired revisions in 
existing compulsory education curricula. More specifically, the design of these new curricula 
has been informed by PISA-like competencies (i.e. skills and knowledge, motivation and 
attitudes that are relevant for life). ’’ 

- Luxembourg - “New school reforms are now being reflected upon for implementation in 
2013. These include definition of achievement standards, new/adapted methods of teaching, 
learning and assessment that emphasise on the measuring how students are able to use what 
is learnt in real-life situations." 



9 . For further details, see PISA 2009 Initial Report, Vol V, Box F: Chile, p 85. 
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iv. Setting national proficiency standards for PISA. 

In Australia and Canada, the national proficiency standards for students in PISA have been set at level 
3 on the international scales. 

3.5. Setting and monitoring performance targets and indicators 

PISA-based national or sub-national performance targets or indicators have been set in 18 
countries/economies, namely: Australia, Belgium (Flemish and French Communities), Canada, Denmark, 
Finland, France, Hungary, Ireland, Israel, Ireland, Japan, Mexico, the Netherlands, Poland, Slovenia, the 
Slovak Republic, Turkey and Wales-UK. Analysis of open-ended responses revealed the diversity of target 
types, number of targets, the stakeholders who set them {e.g. the Prime Minster or a national assessment 
institute), and the documents in which those targets have been elaborated {e.g. whitepaper, growth strategy, 
etc.). The specific country targets are outlined in Table 3.5. Target types included: 

i. Relative rank of country in international performance. These are goals dependent on the relative 
performance of other countries/economies; 

ii. Specific national PISA score, range of scores in each domain, or perform at the OFCD average; 

iii. Increases in number of high-performers and decreases in percentage of students scoring at or 
below PISA proficiency Fevel 2, the baseline proficiency level; and 

iv. Equity goals, including variance between schools. 

Some countries/economies have used a range of target types and indicators. 

In addition to these country/economy performance targets, some responses from European countries 
also emphasised that the council of the EU included a PISA-based target for low achievers within the 
strategic framework for European cooperation on education and training (“FT 2020”). The target states that 
by 2020 the share of pupils performing below PISA proficiency Level 2 should be less than 15%. 



10 . The response from Spain also indicated that national performance targets were set. But the description of these targets 

did not meet the intended definition of targets and thus was excluded from this list. 
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Table 3. 5. Overview of PISA-referenced performance targets and indicators 



COUNTRY 


Targets/Indicators 


Australia 


PISA used as key national performance measures of numeracy, science literacy and literacy. 
These are expressed in the Measurement Framework for Schooling in Australia (ACARA, 2010). 
The proportion of students achieving at or above the proficient standard (defined as Level 3) on 
the OECD PISA combined mathematics, reading and science scale. No specific target is set. 


Belgium - 

(Flemish 

Community) 


The indicator 'Share of 15-year-olds performing at level or below in reading literacy' is used in the 
Flemish 'Pact2020'-a future-oriented action plan that was agreed by all Flemish social partners 
in 2009. The indicator is used to monitor an overarching goal of improving the quality of life and 
reducing poverty. Specifically, the PISA indicator is referred to in the context of literacy. As the 
target concerns literacy of the whole population, no specific target has been set for the PISA 
indicator. 


Belgium 

(French 

Community) 


In the Contrat pour I’ecole^, which is a set of goals assigned to the education system on the 
basis of a broad consultation of the educative community, PISA is mentioned several times. 
Some of the targets for 2013 are linked to PISA: 

- The students’ results in reading, mathematics and science should at least reach and preferably 
surpass the OECD average, and the proportion of weak students should decrease. 

- The between-school variance in student performance should decrease from 56% to 40% and 
the school segregation level should be less than 40%. 

- The variance in results in the main subjects explained by the track should progressively be 
reduced. 


Canada 


Some Canadian provinces have expressed PISA-related performance targets in general terms, 
such as ‘improving PISA score’ or ‘improving PISA ranking’. 


Denmark 


Rank - a goal of Denmark to be in the top five in PISA. The goal was set by the Danish Prime 
Minister in January 2010. 


Finland 


The goal for national performance for each domain, expressed as a range of PISA scores. The 
goal is set in the state budget by the government and approved by the parliament. 


France 


For PISA 2012, the target is 17% for low-achieving readers. 


Ireland 


In the national literacy and numeracy strategy. ° target-setting on the basis of PISA is included as 
follows: 

- Increase the percentage of 15-year-old students performing at or above Level 4 (i.e. at the 
highest levels) in PISA reading literacy and numeracy tests by at least five percentage points by 
2020; 

- Flalve the percentage of 15-year-old students performing at or below Level 1 (the lowest level) 
in PISA reading literacy and numeracy tests by 2020. 


Israel 


Improve Israel’s absolute and relative standing on the PISA scale. Increase in the percent of 
students at the upper part of scales, and reduce the percent of students at the lower end of the 
scales. 


Japan 


The government established the “New Growth Strategy” on 18 June 2010 that included various 
goals to be achieved by the year 2020. PISA is referred to in one of the strategy’s goals: to see 
Japan’s overall student performance at the top level among the participating countries. The aim 
is also to reduce the number of poor performers and increase the number of high performers; 
make the mean scores of each domain close to those of the top-performing countries; and 
increase the proportion of students who respond positively to the attitudes- and interest-related 
items to above-average levels. 


Mexico 


Objective 1 of the Education Sector Programme 2007-2012 establishes as a goal for 2012 to 
raise performance on PISA to 435 points as an average for both mathematics and reading, 
taking the average score of 392 points, attained in PISA 2006, as the base. 

The Agreement for the Articulation of Basic Education asserts that the whole curriculum should 
set a vision for 2021 that includes generalising the competencies described at PISA Level 3; 
eliminating the gap between the students who perform below Level 2 and those who perform at 
or above that level. 



26 





EDUAVKP(2012)8 



Poland 


Polish education strategy seeks to reduce the number of low-achievers in reading, math and 
science as defined by the ET 2020 benchmark. 


Netherlands 


Raise average scores in three PISA domains and increase number of students at highest 
proficiency levels. 


Slovak 

Republic 


The Ministry of Education sets its performance targets in relation to performance of the country 
on PISA. The Ministry, as sole stakeholder, sets the targets with the aim of achieving the OECD 
average or better in the following round. 


Slovenia 


There are no officially set performance targets based on PISA. However, the 201 1 White Paper 
on Education states that Slovenia should rank in the top third of developed countries/economies 
in PISA-like assessments. 


Turkey 


National performance targets are determined according to the country’s score on PISA and the 
OECD average. 


Wales 


The Minister has voiced aspirations for improvement in PISA performance and PISA related 
skills. 



a. This document can be located at: http://www.enseignement.be/index.php?page=25230 

b. Ireland’s national literacy and numeracy strategy document can be found at: 



http://www.education.ie/admin/servlet/blobservlet/lit_num_strat.pdf?language=EN&igstat=true (p18) 

Section 4 Discussion 

This paper has sought to examine the normative impact of PISA by investigating how, and the extent 
to which, national policy actors use PISA in policies and practices to evaluate and improve school-system 
performance. The results make clear that PISA is becoming an influential element of education policy- 
making processes at the national level. Furthermore, the findings provide preliminary evidence that PISA 
is being used and integrated within national/federal policies and practices of assessment and evaluation, 
curriculum standards and performance targets. 

Policy-makers across nearly all PISA-participating countries/economies see PISA as an important 
indicator of system performance, and there is evidence that the PISA evaluation has the potential to 
‘define’ the policy problems and set the agenda for policy debate at the national and state levels. Results 
from the early rounds of PISA triggered large-scale PISA shock in a small group of countries/economies, 
such as Germany and Denmark. This study provides evidence that the majority of countries/economies 
have initiated policy reform and initiatives - to varying extents - in direct response to the PISA evaluation 
at some point across the survey rounds. Countries/economies that have demonstrated the most substantial 
policy responses to PISA included those that perform above, at and below the OECD average. Overall, 
PISA seems to have become accepted by policy-makers as a valid and reliable instrument for 
internationally benchmarking current system performance and the relative changes in outcomes over time. 

Beyond being an external monitor of system performance, there is evidence that PISA has now been 
embedded within national/federal systems for assessment and evaluation. Across a range of 
countries/economies, PISA has provided a large-scale evaluation for systems that did not previously 
conduct national/federal assessments; has promoted the increased scope of national assessment systems; 
and been used to monitor sub-national regions and student groups. In other countries/economies with more 
sophisticated national/federal assessment systems, PISA is used to complement national data and validate 
national results against an international benchmark. There is also evidence that some policy-makers have 
used the PISA assessment frameworks and instrument as a best-practice ‘model’ or ‘guide’ in formulating 
improved national/federal assessment policies and practices. Furthermore, a substantial group of 
countries/economies has explicitly sought to incorporate and emphasise PISA-like competencies in revised 
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national standards and curricula. This impact on assessment and standards may be best interpreted as 
policy transfer and learning from the international to the national level (Dolowitz and Marsh, 2000; Stone, 
2001; 2004). 

There is also evidence that PISA has been embedded as an external global standard for setting system 
goals and evaluating system progress. A substantial number of countries/economies have set PISA-based 
national performance targets. These policy targets often define measurable system goals in terms of 
relative rank or absolute PISA score. Targets focused on improving the performance of low-achieving 
students - defined as below PISA proficiency Level 2 - have been adopted across a broad range of 
countries/economies and at the supranational level by the EU. PISA-based policy targets further 
incorporate PISA norms into the national/federal education system. Eurthermore, respondents from a group 
of countries/economies that have recently enacted, or are in the process of initiating, education reforms 
highlighted the expected role of future PISA surveys in evaluating the effectiveness of these national 
reform strategies. These finding underscore a shift towards international, external accountability for system 
performance and improvement through PISA. 

PISA policy findings have also had an impact on national policy. The PISA-based policy findings 
drawn from the factors associated with higher performance have played an influential role in the policy- 
making processes of the majority of countries/economies, particularly those related to assessments and 
accountability. In the specific area of student selection and tracking, analysis of PISA results has directly 
challenged the historical norm of school organisation in a group of education systems in Europe that 
traditionally track their students. Eurthermore, there is evidence that referring to high-performing PISA 
systems has been influential in the policy-making process of at least half of the countries/economies. The 
education policies of Einland have been cited most broadly across countries/economies, signalling that, 
through its performance in PISA, Einland is regarded as a key source for cross-national policy learning and 
borrowing. It is possible that reference to PISA high-performing countries may help policy actors to 
legitimise and justify their intended reform agenda within contested national policy debates (Steiner- 
Khamsi, 2003; 2010). 

The analysis of this paper suggests that PISA plays an important function for policy-makers as they 
seek to evaluate and improve system performance in response to the demands of the global knowledge 
economy. This paper provides preliminary evidence that PISA has an influential normative effect on the 
direction of national education policies, although the extent of influence varies across countries/economies. 
In some PISA participating countries/economies, policy-makers have made only small policy adjustments 
to respond to the PISA evaluation and align system policy settings with international standards. Yet there is 
evidence that in other countries/economies, policy actors have responded quickly to PISA evaluations of 
their system and have moved to incorporate the PISA norms and standards more deeply into the 
national/federal policy processes and instruments. In these contexts, PISA may come to increasingly shape, 
define and evaluate the key goals of the national/federal education system. On a broader level, the analysis 
discussed in this paper reveals the emerging role and influence of the OECD as an international education 
monitor and policy actor. 
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